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IMAGE PROCESSING METHOD AND APPARATUS, AND STORAGE MEDIUM 



The present invention relates to an image 
processing apparatus and method for reconstructing an 
image, which records image information appended with 
quantized space information in units of pixels such as 
image information expressed by, e.g., a ray space theory, 
in a virtual space and, more particularly, to an 
improvement for assuring a high rendering frame rate by 
lowering resolution as, e.g., motion in the space gets 
faster upon reconstructing an image in a virtual space 
on the basis of recorded image information. 



Attempts to describe a virtual space on the basis 
of a ray space theory have been proposed. See, for 
example, "Implementation of Virtual Environment by 
Mixing CG model and Ray Space Data", lEICE Journal D-11, 
Vol. J80-D-11 No. 11, pp. 3048 - 3057, November 1997, or 
"Mutual Conversion between Hologram and Ray Space Aiming 
at 3D Integrated Image Communication", 3D Image 
Conference, and the like. 
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BACKGROUND OF THE INVENTION 
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A recording method of ray space data will be 
explained below . 

As shown in Fig. 1, a coordinate system 0-X-Y-Z is 
defined in a real space. A light ray that passes through 
5 a reference plane P (Z = z) perpendicular to the Z-axis 
is defined by a position (x, y) where the light ray 
crosses P, and variables 9 and ([) that indicate the 
direction of the light ray. More specifically, a single 

i -I 

in light ray is uniquely defined by five variables (x, y, z, 

13 10 9, ([)). If a function that represents the light 

in intensity of this light ray is defined as f, light ray 

in 

..i group data in this space can be expressed by f (x, y, z, 

■13 

;|U 9, ([) ) . This five-dimensional space is called a "ray 

P 

ij space". 

is 

p 15 If the reference plane P is set at z = 0, and 

disparity information of a light ray in the vertical 
direction, i.e., the degree of freedom in the ([) 
direction is omitted, the degree of freedom of the light 
ray can be regenerated to two dimensions (x, 9) . This x- 
20 9 two-dimensional space is a partial space of the ray 
space. As shown in Fig. 3, if u = tan9, a light ray 
(Fig. 2) which passes through a point (X, Z) in the real 
space is mapped onto a line in the x-u space, said line 
is given by: 

25 X = x + uZ (1) 
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Image sensed by a camera reduces to receiving 
light rays that pass through the lens focal point of the 
camera at an image sensing surface, and converting their 
brightness levels and colors into an image. In other 
words, a light ray group which passes through one point, 
i.e., the focal point, in the real space is captured as 
an image a number of pixels. In this, since the degree 
of freedom in the ^ direction is omitted, and the 
behavior of a light ray is examined in only the X-Z 
plane, only pixels on a line segment that intersects a 
plane orthogonal with respect to the Y-axis need to be 
considered. In this manner, by sensing an image, light 
rays that pass through one point can be collected, and 
data on a single line segment in the x-u space can be 
captured by single image sensing process. 

When this image sensing is done a large number of 
times by changing the view point position, light ray 
groups which pass through a large number of points can 
be captured. When the real space is sensed using N 
cameras, as shown in Fig. 4, data on a line given by: 



can be inputted in correspondence with a focal point 
position (Xn, Zn) of the n-th camera (n = 1, 2,..., N) , 
as shown in Fig, 5. In this way, when an image is sensed 
from a sufficiently large number of view points, the x-u 
space can be densely filled with data. 



X + ZnU = Xj 



•n 



(2) 
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Conversely, an observation image from a new 
arbitrary view point position can be generated (Fig. 7) 
from the data of the x-u space (Fig. 6) . As shown in 
Fig. 7, an observation image from a new view point 
5 position E(X, Z) indicated by an eye mark can be 
generated by reading out data of a line given by 
equation (1) from the x-u space. 
i3 One major feature of ray space data is that ray 

In 

13 space data is defined for each pixel. That is, frame 

13 

10 data for one scene is expressed by ray space data 

m 

::|n corresponding to the number of pixels of that frame. 

■12 Hence, the data size of ray space data does not depend 

;p on the complexity of a scene, but depends on only the 

'p size and resolution of the scene; the computation volume 

15 depends only on the total number of pixels of the scene 

to be generated. In case of normal CG data, when a scene 
becomes more complex, the complexity cannot be expressed 
unless the number of polygons is increased. Hence, the 
computation volume increases, resulting in low rendering 
20 performance. However, in case of ray space data, if the 
total number of pixels of an image to be rendered 
remains the same, rendering performance is constant 
independently of the complexity of scenes. 

A case will be examined wherein the user walks 
25 through a virtual space generated using such ray space 

data. In such case, several ten frames of images per sec 
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must be generated and presented to make the user feel as 
if he or she was walking through the virtual space. 

However^, when the total number of pixels of an 
image reconstructed from ray space data in each frame is 
5 large, a long rendering time per frame is required, and 
the rendering frame rate cannot catch up with the moving 
speed. 

When the user manipulates (moves, rotates, or the 
^L" like) an object in the scene at high speed, and that 



P 

m 
m 



in 
in 



10 object is generated and rendered based on ray space data, 
if the number of pixels of the object is large. 



* rendering cannot be done in time. 

fu 

.|=, SUMMARY OF THE INVENTION 

I 

The present invention has been proposed to solve 
the conventional problems, and has as its object to 
provide an image processing apparatus and method, which 
reconstruct an image by capturing image information at 

20 resolution below that of image information at time of 

recording by lowering the sampling rate when no problem 
is expected even if lowering resolution (e.g., an object 
moving in a space) , upon reconstructing an image from 
image data such as ray space data, image information of 

25 which includes space information, thereby assuring high 
rendering frame rate. 
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In order to achieve the above object, an image 
processing apparatus of the present invention comprises: 

recording means for recording image information 
appended with quantized space information in units of 
5 pixels; 

setting means for setting a sampling rate; 
reconstruction means for reconstructing an image 
by reading out image information from the recording 
means in, accordance with the sampling rate set by the 

-in 

.13 10 setting means; and 

13 

,^ interpolation means for interpolating pixels at 

;tn 

.|n empty pixel positions to obtain a resolution of an image 

.■;t 

,p required for display when the reconstruction means 

■p reconstructs the image at a resolution lower than the 

li 15 resolution of the image required for display. 

■^^ When the sampling rate is determined on the basis 

of, e.g., the moving speed in the virtual space, image 
information is read out at that sampling rate to 
reconstruct an image. When the sampling rate is below 
20 that of the recording resolution, the time required for 

reconstructing an image can be shortened. In addition, a 
decrease in resolution can be compensated for by pixel 
interpolation (e.g., texture mapping). 

A method of the present invention as another 
25 arrangement for achieving the above object is an image 
processing method for reconstructing an image which is 
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recorded as ray space data in recording means according 
to a first resolution, comprising: 

a step of detecting a moving speed in a space of a 
reconstructed image; 
5 a step of determining a second resolution of ray 

space data to be read out from, the recording means in 
accordance with a detected moving speed in the space, 
p and reading out ray space data from the recording means 

in in accordance with the second resolution; and 



id 

13 10 a step of interpolating pixels at empty pixel 

|f| positions produced in correspondence with dropping to 

in 

the second resolution smaller than the first resolution. 

The above object can also be achieved by an image 
processing method or a storage medium that stores a 
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15 program for implementing this method. 

Other features and advantages of the present 
invention will be apparent from the following description 
taken in conjunction with the accompanying drawings, in 
which like reference characters designate the same or 
20 similar parts throughout the figures thereof. 

BRIEF DESCRIPTION OF THE DRAWINGS 

The accompanying drawings, which are incorporated 
25 in and constitute a part of the specification, 

illustrate embodiments of the invention and, together 
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with the description, serve to explain the principles of 
the invention. 

Fig. 1 is a view for explaining the principle of 
generating ray space data; 
5 Fig. 2 is a view for explaining data in a real 

space; 

]3 Fig. 3 is a view showing the space shown in Fig. 2, 

In which is expressed by ray space data; 



13 

m 
in 

..I 



Fig. 4 shows a plurality of cameras to explaining 
10 principle of generating real space data; 

Fig. 5 shows the view of each of a plurality of 
cameras for explaining the principle of generating ray 
space data; 

Fig. 6 is a view for explaining the principle of 



n 
n 

15 reconstructing a real space from ray space data when 



there are a plurality of cameras; 

Fig. 7 is a view for explaining the principle of 
reconstructing a real space from ray space data when 
there are a plurality of cameras; 
20 Fig. 8 is a view for explaining quantization in an 

embodiment of the present invention; 

Fig. 9 is a view for explaining storage of ray 
space data in the embodiment of the present invention; 
Fig. 10 is a view for explaining sampling and 
25 storage of ray space data in the embodiment of the 
present invention ; 
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Fig, 11 is a block diagram showing the hardware 
arrangement of the embodiment of the present invention; 
and 

Fig. 12 is a flow chart showing the control 
sequence in the embodiment of the present invention. 

Fig. 13 is a flow chart showing the process in the 
modification 1 of the present invention. 



:|n DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 

1^ Preferred embodiments of the present invention 

sin 



.13 



will now be described in detail in accordance with the 



;|rt accompanying drawings. 

f =i 

An image processing apparatus and method to which 
15 the present invention is applied will be described in 
detail hereinafter with reference to the accompanying 
drawings. In this image processing apparatus and method, 
when the user walks through a virtual space or 
manipulates (moves, rotates, or the like) an object in 
20 the virtual space expressed by ray space data, the 

rendering frame rate can catch up with the moving speed 
in the space by lowering the resolution by decreasing 
the sampling rate of ray space data, and the decrease in 
resolution is compensated for by pixel interpolation 
2 5 such as texture mapping. 
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Fig. 8 is a view for explaining quantization of 



ray space data of this embodiment. More specifically, 
assume that ray space data to be processed by the image 
processing apparatus of this embodiment is image data 
obtained using a camera which has a CCD having N pixels 
in the main scanning direction, a field angle w, and an 
intersection angle a between the optical axis and Z-axis. 
If the pixels of the CCD are respectively ordered 0,..., 
N-1; 6 represents the angle the light ray which passes 
through the j-th pixel makes with the Z-axis; and x 
represents the position where the light ray which passes 
through the j-th pixel intersects with the X-axis, we 
have : 



Note that mapping in Fig. 8 is done with respect to the 
Y-axis. If equation (3) holds for the j-th pixel on Y ^ 
m-th line (m = an arbitrary number) , the angle 9 
corresponding to pixel j can be computed from equation 
(3). That is, if I"(i) represents the i-th image data, 
and that image data is converted into image data I' in 
an (x, 9) space, which is further converted into image 
data I in an (x, u) space by u = tan9, this process is 
expressed by: 




(3) 



(i) =1' (x, 9) = I (X, u) 



(4) 
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u and X are appropriately quantized. For example, 
as for the x-axis, the spacing between two neighboring 
light rays on the x-axis is obtained, and quantization 
is done using this spacing in a quantization step. Also, 
5 as for the u-axis, quantization is done using the 

tangent of an angle between two neighboring pixels and 
the camera view point position. 

In this manner, ray space data become points 
discretely sampled on a line, as shown in Fig. 10. 
10 Smoothness of movement when the user or object 

moves at high speed in the virtual space depends on the 
rendering frame rate. That is, as the rendering frame 



:|y rate goes higher, the user or object can experience 

vi3 smoother movement in the virtual space. But if the frame 

i3 15 rate is low, the user or object may experience unnatural 

sensation since the low frame rate results in jerky 
motion- Hence, in order to implement smooth movement in 
virtual space when the moving speed of the user or 
object is high, the rendering rate must be higher 
20 improved. However, this means an increase in the total 
number of pixels of an image to be reconstructed from 
ray space data and, hence, rendering may not be done in 
time. To solve this problem, in this embodiment the 
sampling rate in Fig. 10 is changed. That is, if 
25 sampling of every two pixels is fast enough to catch up 
with rendering, every two pixels are sampled. User's 
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moving speed is detected by the movement of a mouse in 
this embodiment. If this speed is assumed to be v cm/s, 
the degree of load on rendering at that time is k.v (k 
is a predetermined constant) . The sampling rate, n, must 
5 assume a highest possible value. The sampling rate n 
(pixel interval) is a function of the load k.v. The 
sampling rate n assumes a numerical value that may 
retard rendering if it is set to be lower than that 

.in 

■i^ value, and can be determined in advance in accordance 

10 with the rendering speed of this system and the value of 
:iin the load k.v. When said n is determined, ray space data 

il are sampled at n-sample intervals in Fig. 10. The data 

ify 

13 size to be processed is decreased by sampling at n-pixel 

intervals to cope with movement of the user or object or 

15 scrolling. Even when the resolution lowers by decreasing 
the data size, while the user or object is moving in the 
space, the visual characteristics of the user who is 
observing that movement, i.e., identification 
performance for details of a moving object or scene 

20 impairs, so a space with lower resolution does not pose 
any problem. 

Fig. 11 shows the system arrangement of this 
embodiment. The hardware arrangement shown in Fig. 11 is 
that of a normal workstation. Referring to Fig. 11, 

25 especially, a disk 25 stores a large volume of ray space 
data, as shown in Fig. 9. This system presents a virtual 
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space to the user on a CRT 23. At this time, the user 
can freely walk through that virtual space or can 
manipulate (move, rotate, or the like) an object in the 
virtual space by operating a mouse 28. The walk through 
speed or manipulating speed of the object is the moving 
speed of the mouse 28. Fig. 12 shows the control 
sequence of the system shown in Fig. 11. 

When the system is started up in step S2, the 
moving direction and moving amount v of the mouse are 
computed on the basis of user operation for the mouse 28 
in step S4 . In step S6, the image size is determined 
from the size of the display range of the CRT 23. In 
step S8, the sampling rate n (pixels) is determined from 
the moving amount v of the mouse, and ray space data are 
read out at this sampling rate n (pixels) from the disk 
25 to reconstruct an image. 

In step SIO, the image is enlarged. Since image 
reconstruction done in step S8 is decimation at the 
sampling rate n, the image is reduced- Hence, the image 
is enlarged to the image size determined in step S6. 
This enlargement is implemented using a texture mapper 
24, a known hardware tool. In this manner, image 
interpolation is done, and image display at the required 
resolution is implemented. 

Various modifications of the present invention may 
be made. 
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Modification 1: 

In the above embodiment, an image is ■ reconstructed 
from a ray space data image database located at the same 
site. Also, the present invention can be applied to a 
case wherein the database is located at a remote 
location and connected via a network. This is because 
when the system of the present invention is used via the 
network, variations of the transfer rate depending on 
the load on the network pose the same problem as that 
posed when the walk through speed of the viewer (user) 
exceeds the rendering rate. That is, even when change or 
motion of an image is small, a jerky image may be 
presented as the frame rate drops via the network. This 
is because the CPU of the receiving image processing 
apparatus cannot receive data fast enough even if it has 
more than adequate capacity. When the present invention 
is applied to solve such problem, by setting the 
sampling rate of image data between the transmitting and 
receiving apparatuses in accordance with the data 
transport rate of the network, a smoothly moving image 
can be provided although image quality deteriorates to 
some extent . 

Fig. 13 shows process of modification 1. When a 
process shown in the Fig. 13 is the same process shown 
in the Fig. 12, the process shown in the Fig. 13 is 
marked the number as the process shown in the Fig. 12. 
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The explanator is omitted. The differences between 
process shown in the Fig. 13 and the process shown in 
the Fig. 12 are judgement the data transport rate of the 
network in step S31, and setting sampling rate of the 
5 image by referring data transport rate in a process of a 
image creating in step S81. 

Modification 2: 
The moving speed can be detected by means other 
than the mouse. That is, the walk through or scrolling 
speed need only be detected. 

In the above embodiment, the load is assumed to 
linearly increase with respect to the moving speed of 
the user. However, the present invention is not limited 
to such assumption. For example, the resolution (the 
total number of pixels of an image) may be set in 
inverse proportion to the square of the speed. 
Furthermore, the resolution may be determined by 
actually measuring the speed. 
Modification 3: 

In the above embodiment, ray space data are 
obtained by computations, but may be pre-stored as a 
table in a RAM or ROM. 
Modification 4: 
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The display device is not limited to the CRT. The 

present invention can also be applied to a lenticular or 

HMD display device. 

To restate, according to the present invention, 

5 even when the virtual space defined by image data such 

as ray space data imposes a heavy processing load on a 

computer, an increase in load can be suppressed by 

P lowering the resolution by decreasing the sampling rate. 

in Especially, when the processing load increases upon 

P 

.|3 10 movement in the virtual space by user operation or 

:;[fj movement of an object in the virtual space, a decrease 

J in resolution by decreasing the sampling rate does not 

pose any problem to user's vision. Since human's visual 
characteristics that perceptual sensitivity to object 
15 moving is low, deterioration of image quality can be 
ingored. 

As many apparently widely different embodiments of 
the present invention can be made without departing from 
the spirit and scope thereof, it is to be understood 
20 that the invention is not limited to the specific 

embodiments thereof except as defined in the claims. 
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